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Abstract 

Background: Health educators need rigorously developed instruments to evaluate cognitive skills relating to 
evidence based practice (EBP). Previous EBP evaluation instruments have focused on the acquisition and appraisal 
of the evidence and are largely based in the medical profession. The aim of this study was to develop and validate 
an EBP evaluation instrument to assess EBP cognitive skills for entry-level health professional disciplines. 

Methods: The Fresno test of competence in evidence based medicine was considered in the development of the 
'Knowledge of Research Evidence Competencies' instrument (K-REC). The K-REC was reviewed for content validity. 
Two cohorts of entry-level students were recruited for the pilot study, those who had been exposed to EBP 
training (physiotherapy students, n = 24), and who had not been exposed to EBP training (human movement 
students, n = 76). The K-REC was administered to one cohort of students (n = 24) on two testing occasions to 
evaluate test-retest reliability. Two raters independently scored the first test occasion (n = 24) to evaluate the inter- 
rater reliability of the marking guidelines. Construct validity was assessed by comparison of the two groups, 
'exposed' and 'non-exposed', and the percentage of students achieving a 'pass' score in each of these groups. Item 
difficulty was established. 

Results: Among the 100 participants (24 EBP 'exposed', and 76 EBP 'non-exposed' students), there was a 
statistically significant (p < 0.0001) difference in the total K-REC scores. The test-retest and inter-rater reliability of 
the individual items and total scores ranged from moderate to excellent (measured by Cohen's Kappa and ICC, 
range: 0.62 to perfect agreement). 

Conclusions: The K-REC instrument is a valid and reliable evaluation instrument of cognitive skills of EBP in entry- 
level student health professionals. The instrument is quick to disseminate and easy to score, making it a suitable 
instrument for health educators to employ to evaluate students' knowledge of EBP or in the evaluation of entry- 
level EBP training. 
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Background 

The most widely accepted definition of evidence based 
practice (EBP) involves three components: the integra- 
tion of the best research evidence with patient values 
and clinical expertise [1]. Evidence based practice is now 
well established in the health and social care professions 



* Correspondence: lucy.lewis@unisa.edu.au 

'School of Health Sciences, University of South Australia, GPO Box 2471, 
Adelaide, SA, 5001, Australia 

Full list of author information is available at the end of the article 



[2]. To date, the main focus of EBP has been on the 
behaviours (application and practice) of health profes- 
sionals in the clinical and research environments [3]. 
More recently, the EBP movement has expanded to 
include the educational processes (content, delivery and 
assessment) involved in entry-level training of health 
professionals [4]. 

It has been proposed that EBP consists of a five step 
process [1,5,6]. Previous authors have suggested that 
this five step model should form the basis not only for 
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health professionals' clinical decision making and prac- 
tice but for curricular content underpinning EBP train- 
ing of health professionals [7]. The first four steps of 
this model involve asking a clinical question, acquiring 
and appraising the evidence, and applying the evidence 
into clinical practice. The fifth step encourages indivi- 
duals to reflect upon the process undertaken in the first 
four steps. This final step may also provide an opportu- 
nity for EBP training providers to undertake formal eva- 
luation procedures to assess the effectiveness of EBP 
training. The recommendations provided in the Sicily 
Statement highlight a need for future educational 
research evaluating EBP training, and the development 
and application of evaluation instruments to assess EBP 
training for each of the five steps [7], 

A previous systematic review [8] identified 104 instru- 
ments for evaluating the effectiveness of EBP training, 
the majority of which were developed or tested with 
medical students or practitioners and the minority (n = 
13) developed or tested on other health professions. 
Seven of the 104 instruments identified in this review 
were recognised as 'Level 1 instruments' (supported by 
established inter-rater reliability, objective outcome mea- 
sures, and three or more types of established validity). 
The 'Fresno test of competence in evidence based medi- 
cine' [9] and the Berlin Questionnaire [10] were the 
only 'Level 1' instruments identified as evaluating all 
aspects of EBP competence [8]. The 'Fresno test' con- 
sists of two clinical scenarios and open ended questions 
to measure medical professionals' skills and knowledge 
across the four main steps of the EBP process. The Ber- 
lin Questionnaire consists of 15 multiple choice ques- 
tions which mainly focus on participants' 
epidemiological skills and knowledge. While the Berlin 
questionnaire is described as an instrument that assesses 
EBP competence [8], it only comprehensively evaluates 
the third step (appraisal) of the EBP process [11]. 

A further 10 instruments evaluating cognitive EBP 
skills have been published since the 2006 [8] systematic 
review (Table 1) [12-21]. Four of these instruments were 
designed for disciplines external to medicine including 
complementary medicine [13], nursing (diabetes educa- 
tors) [15], occupational therapy [20] and physical ther- 
apy [21]. 

At the time of the current study (2008), the instru- 
ment developed in this study, the 'Knowledge of 
Research Evidence Competencies' instrument (K-REC) 
was the first known tool prospectively designed to col- 
lect data on cognitive EBP skills of entry-level allied 
health students. Since the initial literature search and 
development of the K-REC, two further instruments 
have been published that evaluated cognitive skills of 
EBP in the allied health disciplines [20,21]. The 
Adapted Fresno test of competence in evidence-based 



practice' (AFT) was developed for occupational thera- 
pists, based on the first seven items of the Fresno test. 
Inter-rater reliability (ICC 0.91-0.96) and internal con- 
sistency (Cronbach's a 0.74) was confirmed with a sam- 
ple of 10 occupational therapists representative of the 
wider sample studied (75% graduated > 5 years) and the 
AFT demonstrated responsiveness before and after a 
two day EBP workshop [20]. The 'Modified Fresno test' 
(MFT) for physical therapists considered all 12 items of 
the Fresno test, with item stems almost identical to the 
original test [21]. Two further items were added to the 
modified test regarding the other pillars of EBP; patient 
preferences and clinical expertise, with the clinical 
expertise item removed following poor psychometric 
performance. Three cohorts of respondents were 
recruited for this study, first year physical therapy stu- 
dents (novices n = 31), third year Doctor of Physical 
Therapy students (EBP trained students n = 50) and 
physical therapy faculty (EBP experts n = 27). The 
'Modified Fresno test' for physical therapists demon- 
strated inter- and intra-rater reliability (inter-rater ICC 
0.91, intra-rater ICC 0.95, 0.96), internal consistency 
(Cronbach's a 0.78) content and discriminative validity 
(significant difference in total score corresponding to 
training level p < 0.0001) [21]. 

While the Fresno test has been recognised as the cur- 
rent best available instrument to comprehensively assess 
cognitive skills of EBP [8], the instrument is not without 
limitations as noted by developers of two modified ver- 
sions [20,21]. The Fresno test includes open ended 
questions requiring short essay style answers. The 
answers are then assessed using a standard scoring tem- 
plate with points awarded for key components of each 
answer. The authors of the Fresno test estimate that 
respondents require up to 60 minutes for completion of 
the test [9]. The test is therefore lengthy to complete 
and difficult to score, with training necessary to score 
the test. The original test was not relevant to health 
professions outside of medicine, with medical based 
clinical scenarios, and reference to issues of diagnosis 
which would be unlikely to be raised in other health 
professions [11]. The original Fresno test has been mod- 
ified for single profession use in occupational therapy 
[20] and physical therapy [21]. Both of the modified 
Fresno tests have retained the open ended style of ques- 
tions and complex scoring template of the original test. 
Given the absence of an instrument specifically devel- 
oped for students across entry-level health professional 
programs, there was a need for the development of an 
instrument based on the Fresno test as a means to 
assess cognitive EBP skills across a range of entry-level 
health professional disciplines. 

Due to the focus of this study on the assessment of 
entry-level education (novice rather than postgraduate 
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Table 1 EBP evaluation instruments identified in an updated systematic search (post 2006) and comparison to the 
Fresno test 

Study Instrument No. of Psychometric EBP EBP content assessed EBP 

type/name knowledge properties outcomes steps 
items 

(question Research Search Research Critical Statistics Levels 

type) question strategy design appraisal of 

evidence 



Ramos et 
al 2003[9] 


Fresno test 


12 (short 
answer/ 


Inter-rater 
reliability, IC, 

rnntpnt 

discriminative 
and 

responsive 
validity 


Ka 


X 


X 


X 


X 


X 




1,2,3,4 


Caspi et al 

2006 [1 2] 


online survey 


10 (m/c) 


IC content 
validity 


K« Kcd A 

K A< K SR< " 










X 




3 


Krueger 
9006 n 31 


exam 


not 

rpnnrtprl 

(m/c) 


Content and 

H isrri m i native 

LJlJl^l II 1 III CI LI V CT 

validity 


Ka 






X 


X 


X 


X 


3 


Novack et 
al 2006 [14] 


survey 


not 

reported 
(m/c) 


Content and 
discriminative 

val iri itv 

VOI HJI LV 


Ka 






X 




X 




3 


Meyer et al 
2007 [15] 


not reported 


13 (not 
reported) 


Inter-rater 
reliability, 

rnntpnt snH 

1 1 ICT ML □ 1 1 

responsive 
validity 


Ka 






X 




X 




3 


Shuval et al 
2007 [16] 


written 
assignment 
-F online 
exam 


not 

reported 
(short 
answer/ 
essay) 


Inter-rater 
reliability, 
content va idity 


K a , A, U, B 


X 


X 










1,2 


Siriwardena 
et al 2007 
[17] 


Manchester 
short EBM 
survey 


30 (m/c) 


IC, content and 

discriminative 

validity 


Ka, A 




X 




X 






2, 3 


Davis et a 
2008 [1 8] 


survey 


5 (m/c, 

short 

answer) 


IC, content, 
discriminative 
and responsive 
validity 


Ka, A 








not reported 








Ahmadi- 
Abhari et al 
2008 [1 9] 


survey 


6 (m/c) 


IC, content and 

discriminative 

validity 


Ka, A 








not reported 






3 


McCluskey 
& Bishop 
2009 [20] 


Adapted 
Fresno test 
of 

competence 
in EBP 


7 (short 
answer/ 
essay) 


Inter-rater 
reliability, IC 
and responsive 
validity 


Ka 


X 


X 


X 


X 


X 




1,2,3 


Tilson 2010 
[21] 


Modified 
Fresno Test 


13 (short 

answer/ 

essay) 


Inter- and intra- 
rater reliability, 
IC, content and 
discriminative 
validity 


Ka 


X 


X 


X 


X 


X 




1,2,3, ? 
4 



Bold type indicates the original Fresno test [9], 'x' research evidence competency assessed, ? questionable inclusion. 

A Attitudes, B Barriers, IC Internal consistency, K A actual knowledge, K SR self-reported knowledge, m/c multiple choice, P Perceptions, T/F true/false, U Use. 



learners or clinicians), the instrument was intended to 
evaluate the first three fundamental steps of the EBP 
process model (ASK, ACQUIRE and APPRAISE), 
rather than assessment of the fourth step involving the 
application of knowledge and skills in clinical practice. 
The aim of this study was to develop an instrument to 



evaluate entry-level respondents' cognitive skills 
regarding the research evidence component of EBP 
that: 

• could be used across the health professions; 

♦ was quick to complete and easy to score; and 
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♦ was underpinned by a transparent and defensible 
psychometric testing process. 

Method 

The development of the instrument was completed in 
two stages. The first stage involved the development of 
the instrument and the second stage comprised the pro- 
cesses used to psychometrically evaluate the instrument. 

Ethical approval for the study was obtained from the 
University of South Australia Human Research Ethics 
Committee (protocol number P067/08). 

Stage 1: Development of the instrument 

Firstly, the research team considered the content of the 
Fresno test in the drafting of the instrument. The first 
seven items of the Fresno test were relevant for applica- 
tion to health professions external to medicine and were 
considered in the development of the K-REC instru- 
ment. The remaining Fresno items (8-12) involved 
advanced statistical calculations (eg predictive values, 
number needed to treat, risk reduction), and questions 
about diagnosis and prognosis. While these skills are 
pertinent to all health professions, the instrument was 
intended for use with entry-level health professional stu- 
dents with varying degrees of exposure to EBP educa- 
tion (novice learners rather than graduates of 
professions). Therefore, a choice to focus on what might 
reasonably be expected to be taught generically across a 
range of degrees/stages of programs and across disci- 
plines was required. In both forms of the modified 
Fresno test [20,21], the specific statistical items of the 
original Fresno test have been altered or removed. The 
choice was made to limit statistical knowledge items in 
the instrument to interpretation of metrics of signifi- 
cance and focus upon questions of intervention. Knowl- 
edge of levels of evidence was not assessed in the 
original Fresno test but was considered an essential 
component and was therefore included. 

One clinical scenario template was designed for the K- 
REC, with specific inclusion of each of the PICO com- 
ponents. The clinical scenario needed to be relevant to a 
variety of health professional disciplines and was there- 
fore deliberately designed around the topic of a chronic 
condition (cystic fibrosis) and two possible types of 
intervention for this condition. The interventions 
included in the scenario (exercise and breathing exer- 
cises) were intentionally chosen to be understood by a 
number of health professional disciplines (e.g. phy- 
siotherapy, occupational therapy, podiatry, human 
movement). 

Given the time and scoring related limitations of the 
Fresno test, the K-REC was designed to be relatively 
quick for respondents to complete and easy to score. 
The K-REC therefore consisted of a combination of 



short answer, multiple choice and true/false type ques- 
tions rather than conforming to the open ended short 
essay answer style of the Fresno test. The final draft of 
the K-REC instrument was distributed to three senior 
academics with expertise in research methods and EBP 
and two entry-level physiotherapy students at the Uni- 
versity of South Australia to review the clarity and for- 
matting of the items and instructions. The instrument 
underwent three rounds of review and minor revision to 
the wording, instructions and layout of the instrument. 
The final K-REC instrument 

The K-REC instrument was designed to evaluate cogni- 
tive skills of EBP, with a combination of items evaluat- 
ing either knowledge, or skills in the application of 
knowledge (for example, the application of knowledge to 
write a relevant research question based on the clinical 
scenario). The K-REC instrument was intended to assess 
entry-level respondents' cognitive skills regarding the 
research evidence component of EBP. The instrument 
therefore covered the first three steps of the five step 
EBP process model relevant to cognitive skill (ASK, 
ACQUIRE and APPRAISE), but did not consider the 
last two steps involving the application of evidence into 
practice (APPLY and ASSESS) [7]. The final draft of the 
K-REC consisted of a clinical scenario template and 
nine items relating to that scenario (Additional file 1). 

The 'correct' answers for each item of the K-REC were 
determined through review of the Fresno test, and con- 
sultation and consensus of the research team and senior 
academics during the development procedure. The two 
open-ended items (1 and 7) were scored according to 
the presence or absence of set criteria. For item one, 
respondents were allocated half a mark for each of the 
PICO (Population, Intervention, Control or Comparator, 
Outcome) components provided in their question. In 
item seven, respondents were required to list four char- 
acteristics of randomised controlled trials that would 
increase their confidence that the research was metho- 
dologically sound. Half a mark was awarded for any 
characteristic that was consistent with the Physiotherapy 
Evidence Database (PEDro) scale [22] which is an 
appraisal of methodological bias tool specifically for ran- 
domised controlled trials. Two other characteristics of 
randomised controlled trials were added as correct 
answers for this item as a result of answers provided by 
students during the survey development procedure 
(refer to the K-REC marking guidelines in Additional 
file 2). 

Stage 2: Pilot study: psychometric evaluation of the K- 
REC instrument 

The aims of Stage 2 of the instrument development pro- 
cess were to explore the practicalities of the survey pro- 
cess (recruitment, dissemination, workload required of 
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participants, and response rate), establish test-retest and 
inter-rater reliability, investigate the validity of the K- 
REC, and seek informal feedback from the participants 
about the appearance, layout and general user friendli- 
ness of the instrument. 
Participants 

The recruitment strategy aimed to enlist two main types 
of participants: 

♦ 'Exposed': participants who were representative of 
the target population for the K-REC and who could be 
reasonably expected to have a good understanding of 
EBP through exposure to formal EBP training, and, 

♦ 'Non-exposed': participants who could be reasonably 
expected to have minimal understanding of EBP through 
minimal or no exposure to formal EBP training. 

Third year physiotherapy students (n = 24) were 
invited to participate as the 'exposed' group. At the time 
of the study (2008), these students had completed either 
a mandatory stand-alone EBP course (13 weeks) or an 
Honours research preparation course (2 weeks and 
ongoing supervision within a research team) where they 
were taught and assessed on the different components 
of EBP. Therefore, these students had prior exposure to 
formal training and were expected to have a foundation 
knowledge and understanding of EBP. 

Human movement students (n = 89) were invited to 
participate as the 'non-exposed group'. Rather than 
compare first and third year students from the same 
professional discipline (physiotherapy), human move- 
ment students were selected as they would have been 
exposed to similar information technology literacy (data- 
bases, searching) and were at the same time point in 
their entry-level programs as the physiotherapy students 
(third year). The human movement students were 
expected to have minimal knowledge of EBP, as at the 
time of the study these students were not required to 
enroll or be assessed in any EBP or research training 
course and were therefore not exposed to any formal 
standardised EBP or research instruction. 
Pilot study procedure 

The entire group of 24 physiotherapy students was 
invited to complete the K-REC instrument at a face to 
face class. Students were requested to write their stu- 
dent identification number on the survey to be used to 
match the second round of surveys (test-retest). A sec- 
ond administration to the participating physiotherapy 
students (n = 24) was completed three days after the 
first testing occasion. The completed surveys from the 
first testing occasion (n = 24) were independently scored 
by two raters to establish the inter-rater reliability of the 
instrument. In order to establish the discriminative 
validity of the K-REC, the 'non-exposed' cohort of 
human movement (n = 89) students were invited to par- 
ticipate at a face to face class. The mean item and total 



K-REC scores of each of the student groups ('exposed' 
and 'non-exposed') were compared to establish discrimi- 
nant validity. 
Data analysis 

Test-retest and inter-rater reliability was assessed by 
using the intraclass correlation coefficient (ICC) for 
items with interval data or those with ordinal data 
where the intervals between measurements were 
assumed as equivalent (item 7 and total scores). The 
reliability of the remaining items was assessed using 
Cohen's Kappa coefficient (un-weighted) for categorical 
data (items 2, 4, 5, 6, 8 and 9) and percentage agree- 
ment (items 1 and 3). Agreement of less than 0.50 was 
classified as poor, between 0.50 and 0.75 as moderate, 
and greater than 0.75 was classified as a good level of 
reliability [23]. 

Discriminative validity was assessed by comparison of 
the individual item and total scores from the two stu- 
dent groups (human movement and physiotherapy) with 
calculation of unpaired t tests, and the z-test for com- 
paring proportions (two-tailed) for the differences 
between the percentage of students achieving a pass (> 
50% score) mark for each of the survey items in both 
student groups [24] . Probability values of less than 0.05 
were deemed statistically significant. 

Results 

The K-REC achieved an overall response rate of 88 per 
cent (physiotherapy students: 100%, human movement 
students: 85%). The average completion time for the K- 
REC was approximately 10 minutes. 

Reliability 

All K-REC items and the total score achieved a moder- 
ate level of test-retest reliability or above (Table 2). All 
participating 'exposed' participants (n = 24) completed 
the retest. A good level of agreement between raters 
was achieved for the individual items and the total score 
(Table 2). 

Validity 

Discriminative validity of the K-REC was determined 
by the comparison of two student groups, one group 
with prior exposure to EBP training (expected to have 
an understanding of EBP process), and one group with 
no prior exposure to formal EBP training (expected to 
have minimal understanding of EBP process). The 
results of the total and individual survey item scores of 
the K-REC for both groups are shown in Table 3. The 
mean total score was 4.4 points higher for the 
'exposed' students compared to the 'non-exposed' stu- 
dents (8.4 versus 4.2 respectively. Unpaired t tests 
were performed to investigate the differences between 
these groups. Seven out of 10 K-REC items (items 8a 
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Table 2 Test-retest and inter-rater reliability of the K-REC 
instrument 



Item no. and content 
assessed 


Test-retest 
reliability 


Inter-rater 
reliability 


1 Research question (PICO) 100% 100% 


2 Sources of information 


K 0.91 (0.85 ■ 


0.97) 


K 1.00 


3 Study design knowledge 


1 00% 




96% 


4 Search strategy (MeSH) 


K 0.93 (0.79 - 


1 .00) 


K 1.00 


5 Search strategy (Boolean) 


K 0.77 (0.54 - 


1 .00) 


K 0.83 (0.44 - 1 .22) 


6 Critical appraisal 


K 0.71 (0.32 - 


1 .00) 


K 1.00 


7 Critical appraisal 


ICC 0.8C 


) 


ICC 0.87 


8a Research evidence 
statistics 


K 0.86 (0.68 - 


1 .00) 


K 1.00 


8b Research evidence 
statistics 


K 0.94 (0.82 - 


1 .00) 


K 1.00 


9 Levels of evidence 


K 0.62 (0.32 ■ 


0.92) 


K 1.00 


K-REC total scores 


ICC 0.8? 




ICC 0.97 



ICC Intraclass correlation coefficient, K Cohen's Kappa coefficient (95% 
confidence interval), MeSH Medical subject headings, PICO {Participants, 
Intervention, Control or Comparator, Outcome) 



and 8b were considered separately) showed a signifi- 
cant difference (questions 1, 3, 6, 7, 8a, 8b and 9). The 
three items (questions 2, 4 and 5) that did not show a 
significant difference were all related to search strat- 
egy. There was a highly significant difference (p < 



0.0001) in the mean total K-REC scores between the 
'exposed' and 'non-exposed' students. 

Considering the mean scores of each group of stu- 
dents for each K-REC item, for nine out of 10 (item 
eight has two sections which were considered separately) 
instrument items a higher proportion of 'exposed' than 
'non-exposed' students passed. A z-test for comparing 
proportions (two-tailed) was performed to investigate 
the differences between the groups resulting in a statisti- 
cally significant difference between groups for seven of 
the 10 instrument items (Table 3). 

The values presented in Table 3 of the percentage of 
pilot study participants who achieved a pass mark (>50% 
score) for each K-REC item and the total scores were a 
representation of the relative difficulty of each of the 
items. The relative difficulty of each item ranged from 
'difficult' with only 26 per cent of the total pilot study 
participants passing the first section of the research evi- 
dence statistics question (8a) and 15 per cent passing 
the second section (question 8b), to 'moderate' with 84 
per cent of participants achieving a pass mark for ques- 
tion two on the identification of sources of information. 

Discussion 

The K-REC met the basic psychometric requirements 
for reliability, validity and usability of a standard 



Table 3 Comparison of scores obtained from the 'exposed' and 'non-exposed' groups and the percentage of students 
who achieved a >50% pass mark for each K-REC item 



Item no. and content 


Max. possible 
score 


Student scores 
mean (SD) 


p value (unpaired t 
test) 


Percentage of students 
passing (>50%) 


p value 
(z score, 2 
tailed) 






'Exposed' 
n = 24 


'Non- 
exposed' 
n = 76 




'Exposed' 
n = 24 


'Non-exposed' n 
= 76 




1 Research question 
(PICO) 


2 


1 .5 (0.0) 


1.1 (0.6) 


0.0001 


100 


78 


0.005 


2 Sources of information 


2 


1 .3 (0.3) 


1 .2 (0.5) 


0.2729 


100 


79 


0.006 


3 Study design 


1 


1 .0 (0.0) 


0.6 (0.5) 


< 0.0001 


100 


55 


< 0.0001 


4 Search strategy (MeSH) 


0.5 


0.3 (0.3) 


0.3 (0.3) 


1 


50 


50 


1.00 


5 Search strategy 
(Boolean) 


0.5 


0.3 (0.3) 


0.2 (0.2) 


0.1693 


54 


38 


0.08 


6 Critical appraisal 


1 


0.8 (0.4) 


0.1 (0.4) 


< 0.0001 


79 


14 


< 0.0001 


7 Critical appraisal 


2 


1 .8 (0.4) 


0.4 (0.7) 


< 0.0001 


100 


24 


< 0.0001 


8a Research evidence 
statistics 


1 


0.4 (0.5) 


0.2 (0.4) 


0.0453 


42 


21 


0.018* 


8b Research evidence 
statistics 


1 


0.4 (0.5) 


0.1 (0.3) 


0.0003 


38 


8 


< 0.0001 


9 Levels of evidence 


1 


0.7 (0.3) 


0.2 (0.3) 


< 0.0001 


79 


26 


< 0.0001 


Mean total score 


12 


8.4 (1.4) 
(range 6.5 - 
11.0) 


4.2 (2.0) 
(range 1.0 - 
11.5) 


< 0.0001 


100 


21 


0.005 



MeSH Medical subject headings, PICO (Participants, Intervention, Control or Comparator, Outcome) 
p values shown in bold type represent a statistically significant difference (p < 0.05) 
* not significant after Bonferroni correction 
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instrument for collecting data on respondents' cognitive 
skills of EBP. The information gained through testing 
the instrument on two health professional discipline 
areas at one institution suggests that if the instrument is 
disseminated in a similar fashion, a reasonable response 
rate should be achieved. 

There are a variety of instruments available to assess 
EBP cognitive skills, some of which have been demon- 
strated to discriminate between novice and expert lear- 
ners. The issue with pre-existing tools is that for novice 
learners, the relevance of the scenario questions and a 
floor effect is likely to be the issue especially with entry- 
level students in the early years of their programs. The 
K-REC instrument was designed to assess the cognitive 
EBP skills of novice learners (entry-level students) irre- 
spective of their professional discipline. That is, the 
anticipated target audience of this instrument was pre- 
dominantly entry-level health professional students, who 
could range from pre-novice (no exposure to EBP prin- 
ciples at all), novice and intermediate EBP learners. 
While the K-REC could be used for expert learners, we 
suspect that a ceiling effect is likely to occur which may 
make it a useful tool for comparison between partici- 
pants of varying EBP exposure but it is likely to be 
insensitive to change in EBP experts (which is likely to 
be the case for most, if not all other EBP knowledge 
instruments). 

The K-REC differed from existing instruments in 
terms of design, length, content and psychometric prop- 
erties. The K-REC contains a clinical scenario template 
that may be relevant to a variety of health professional 
disciplines and individual items relating to the scenario. 
The K-REC therefore has the potential to be applicable 
to a variety of health disciplines, rather than being speci- 
fic to one discipline. In contrast to the original Fresno 
test [9], and two modified versions [20,21], the K-REC 
was designed to be short, easy to complete and dissemi- 
nate. The short answer and multiple choice design 
ensured the usability of the instrument as reflected in 
the short completion times. 

The K-REC demonstrated test-retest and inter-rater 
reliability, content and discriminative validity (Table 4). 
A follow on study [25] established the responsive valid- 
ity of the instrument, with the K-REC demonstrating 
the ability to detect impact of EBP training in a cohort 
of 77 physiotherapy students (p < 0.001, effect size 
1.13). When comparing the psychometric properties of 
the K-REC to the other two instruments developed for 
allied health (Table 1), the K-REC was the only instru- 
ment that established test-retest reliability. The multiple 
choice and short answer design of the instrument, and 
the set marking guidelines are likely to have resulted in 
the high level of inter-rater reliability of the instrument 
(range: 0.83 to perfect agreement). The K-REC was able 



to effectively discriminate between those who had prior 
formal exposure to EBP training and those who had 
minimal exposure. Due to the lack of a comparator 
instrument which assessed cognitive EBP skills in the 
allied health disciplines at the time of the study, conver- 
gent validity was unable to be assessed. The subsequent 
publication of the two modified Fresno tests developed 
for the allied health disciplines [20,21] provides the ideal 
opportunity to compare the K-REC to these alternate 
instruments. 

Developers of instruments assessing EBP knowledge 
face a number of philosophical and practical issues. 
While the Sicily Statement clearly conveys the five fun- 
damental steps of EBP, no similar consensus statements 
exist to guide which specific knowledge and skills might 
be considered essential to all learners regardless of stage 
of training (novice or expert) or professional discipline 
versus optional, advanced or profession specific knowl- 
edge. This might explain why there is such a profusion 
of derivative instruments from the original Fresno test 
which modify, exclude or add items concerning statisti- 
cal concepts, scenarios relevant to various professional 
groups, or focus upon questions of intervention rather 
than giving equal weight to questions of prognosis or 
diagnosis. Where instruments have been developed to 
assess the learning outcomes of specific EBP training 
courses, it is likely that the key items included within 
the assessment are reflected within the specific EBP 
education program. For example, the content of the 
Fresno test [9] was designed "to assess the effectiveness 
of a comprehensive evidence based medicine curriculum 
in the University of California, San Francisco's Fresno 
family practice residency programme" (pg 319) and as 
such, the content of the instrument reflected the learn- 
ing objectives of a specific evidence based medicine 
course reviewed and revised by family practice residents, 
faculty and self identified evidence based medicine 
experts. In contrast, the Adapted Fresno Test [20] was 
developed to assess change in skills and knowledge after 
completion of a two day EBP workshop for occupational 
therapists. The curriculum of the workshop did not 
include number needed to treat, risk reduction or how 
to interpret studies of diagnostic accuracy, which were 
considered to be more suitable for advanced workshops. 

While it is clear that entry-level students should be 
able to critically analyse and evaluate the usefulness of 
research in disciplines such as physiotherapy and occu- 
pational therapy [26,27], there is currently no consensus 
about what statistical content should be included in 
entry-level health professional curricula. In medicine, it 
has been clearly documented that the processes used to 
teach, facilitate learning and assessment of medical sta- 
tistics should be student centred and clinically inte- 
grated [28,29]. However, there are currently no clear 
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Table 4 Summary of the psychometric properties of the K-REC* 



Test property 



Measure used 



Acceptable results 



K-REC performance 



Test-retest reliability 

(comparison of scores on an 
initial test to scores by the 
same participant on a retest) 
Inter-rater reliability 

(the degree of agreement 
between two independent 
raters) 



Cohen's Kappa, ICC and 
percentage agreement 



At least a moderate level of 
agreement (0.50) between testing 
occasions (test-retest) or raters 
(inter-rater) 



Test-retest reliability 

Each of the items and total scores achieved at 
least a moderate level of agreement (range: 
0.62 to perfect agreement) 
Inter-rater reliability 

Each of the items and total scores achieved a 
very good level of agreement between raters 
(range: 0.83 to perfect agreement) 



Content validity (instrument 
covers entire topic of 
interest) 


Expert opinion 


Test covers all of the main 
aspects of EBP 


Content and revisions based on experts' 
suggestions 


Item difficulty (relative 
difficulty of each item) 


The % of candidates who 
answer achieve a passing score 


A wide range of difficulties allows 
a test to be used with both 
'exposed' and 'not exposed' 
groups 


Ranged from moderate (84% question 2) to 
difficult (15% question 8b) 


Construct validity (evidence 
that the instrument measures 
the construct that it intends 
to) 


Discriminative validity 

Mean scores of 'exposed' and 
'not-exposed' compared by f 
test 


Significant difference, higher 
'exposed' student scores 


'Not exposed' mean (human movement) was 
4.2 and 'exposed' mean (physiotherapy) was 8.4 
(p < 0.0001) 



% passing for 'exposed' and Higher % of 'exposed' students For 9 out of 10 instrument items a higher 
'not-exposed' groups compared passing proportion of 'exposed' (physiotherapy 

by the z-test for comparing students) than 'not exposed' (human 

proportions movement students) passed. There was a 

significant difference between groups for 7 of 
the 10 instrument items. 



EBP Evidence based practice 

* Responsive validity established in a follow on study [25], with the K-REC demonstrating the ability to detect impact of EBP training in a cohort of 77 
physiotherapy students (p < 0.001, effect size 1.13). 



guidelines about exactly what content should be taught 
in undergraduate medical programs, with studies on sta- 
tistics training showing differing content between insti- 
tutions [30,31]- Our intent in developing the K-REC was 
to include items which could be reasonably expected to 
be included within entry-level curricula relating to EBP 
education regardless of year level across a range of 
health professional programs, the majority of which are 
targeted at providing health care interventions. Conse- 
quently rather than include more advanced statistics 
(risk ratio, number needed to treat) we opted to include 
metrics of significance (p value, effect size and confi- 
dence intervals) and questions relating to intervention 
(rather than prognosis/diagnosis). 

An additional issue concerning development of EBP 
evaluation instruments is whether the instrument is 
intended for repeated use with the same group of parti- 
cipants (eg pre and post EBP training). Where the iden- 
tical instrument is used on repeated occasions with the 
same group, concerns arise over respondent familiarity 
with the items and expected answers. In a number of 
instruments, provision has been made for alternate 
items to be used on repeated occasions (Berlin: Set A or 
B, APT: two clinical scenarios, MFT: three clinical sce- 
narios). The clinical scenario for a question of interven- 
tion developed for the K-REC was intended to comprise 
a template where the specific condition, intervention, 



comparator and outcome could be modified without 
altering the intent of the questions relating to the sce- 
nario (Ql: write a PICO format search question, Q2: 
identify sources of information likely to provide reliable/ 
valid information Q3: most appropriate design to answer 
search question etc). For example, the PICO compo- 
nents relevant in the scenario included within this tested 
version of the K-REC were: 

"Jane is a 16 year old girl who has cystic fibrosis 
(POPULATION) and she has recently been admitted 
to hospital with a chest infection. Jane normally self 
treats at home with breathing exercises (INTERVEN- 
TION) taught to her by a physiotherapist. One of her 
friends also has cystic fibrosis (POPULATION) but she 
treats herself with exercise (COMPARATOR), not 
breathing exercises. Jane wants to know whether her 
lung condition (OUTCOME) would be more effectively 
managed with an exercise program. You have no experi- 
ence of either breathing exercises or exercise programs 
for cystic fibrosis and are not sure what to recommend." 

This scenario could be modified to be relevant to any 
health professional discipline by altering the PICO com- 
ponents, for example, based on one of the clinical sce- 
narios within the AFT [20], the scenario could be 
modified to: 

"Jane is a 16 year old girl who has a traumatic brain 
injury (POPULATION), currently completing her final 
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year exams and referred to you for memory rehabilita- 
tion. Jane has a number of usual strategies (INTER- 
VENTION) but one of her friends also has had a 
traumatic brain injury (POPULATION) but she man- 
ages her memory issues with a diary (COMPARATOR). 
Jane wants to know whether she would function better 
at school (OUTCOME) with a diary. You have no 
experience of the usual strategies or diaries for memory 
problems in traumatic brain injury and are not sure 
what to recommend." 

When the K-REC instrument with the above alternate 
clinical scenario was completed by a sample of entry- 
level physiotherapy students (n = 15), the inter-rater 
reliability of the marking guidelines remained high (2 
raters, item 1: 93% agreement, items 2-6 and 8-9: 100%, 
item 7: ICC 0.97, and total scores: ICC 0.97). The K- 
REC currently consists of a clinical scenario template, 
with two 'tested' scenarios. If the instrument were to be 
applied longitudinally, it would be necessary to develop 
and test a 'bank' of alternate clinical scenarios. While 
the option is there for users of the K-REC instrument to 
modify the clinical scenario template to suit various 
health professional disciplines, it is important to note 
that any modified scenario would need to be tested 
appropriately. It is also worth considering that the clini- 
cal scenarios contained in the original Fresno test, two 
modified versions and the instrument developed in the 
current study were all based on intervention style ques- 
tions. The Centre of Evidence Based Medicine Levels of 
Evidence [32] currently recognises six different types of 
EBP questions (pre-test probabilities, diagnostic accu- 
racy, prognosis, treatment benefits, treatment harms, 
and screening). While the original and the modified 
Fresno test for physical therapy [9,21] included two final 
open ended items unrelated to the scenarios relating to 
the identification of the best study design for questions 
of diagnosis and prognosis, it is interesting to note that 
none of the instruments contained clinical scenarios or 
the majority of items relating to different types of 
questions. 

Limitations 

It is possible that the three days between test occasions 
resulted in a degree of respondent familiarity with the 
items in the K-REC instrument for the test-retest relia- 
bility testing. The choice of how long to wait before 
retesting knowledge items requires a trade-off between 
the likelihood of participant recall and learning. In test- 
retest situations where the intent is to determine stabi- 
lity of respondent's answers, a longer duration is likely 
to reduce the potential for participants to simply recall 
the answers they provided during the first test occasion 
but increases the possibility of ongoing learning of the 
specific knowledge concepts included in the original 



test. In the current study, participants were university 
students currently engaged in courses where it was likely 
that there would be ongoing exposure to EBP principles 
and knowledge. The choice of a shorter period between 
test occasions was selected as a compromise between 
recall and learning. 

The predominant multiple choice question design of 
the K-REC has both advantages and disadvantages. By 
providing a list of possible answers, it may encourage 
respondents to complete the question (rather than omit 
the question) due to the fact that the 'correct' answer is 
somewhere in the list of options. The answers in the list 
may also act to prompt respondents if they are unsure 
or having difficulty remembering the answer to a parti- 
cular question. It is possible that if the same items were 
designed as 'fill in the blank' type questions, respondents 
may be less likely to guess the answer. It is not possible 
to know from the data collected in this study whether 
respondents were influenced by the provision of answers 
in the multiple choice items in the K-REC. 

The items in the K-REC that related to the second 
step of the EBP process (ACQUIRE) did not discrimi- 
nate between the groups with differing exposures to for- 
mal EBP training (items 2, 4 and 5). The ability to 
effectively search the literature is an essential step in the 
EBP process [33]. Identifying sources of information 
such as databases and journals, and completing an effi- 
cient and effective search strategy requires skills and 
knowledge that are applicable to most areas of health 
professional training, rather than being specific to EBP. 
It is likely that the students involved in this study (third 
year students), had previously obtained skills and knowl- 
edge in searching the literature effectively in earlier 
years of their entry-level programs (eg for assignments), 
regardless of the presence or absence of formal EBP 
training. It is also possible that the true/false design of 
items 4 and 5 may have resulted in an inability of these 
items to discriminate between leaners who 'know about 
the concept' of search strategy, as opposed to learners 
who 'know how to apply' the concept [34]. As the K- 
REC instrument was designed to evaluate skills and 
knowledge of EBP, rather than behaviours (actual per- 
formance of EBP in practice), the underlying reason for 
the lack of discrimination found in these items is 
unclear. 

Implications 

The K-REC instrument was designed to evaluate the 
EBP cognitive skills of novice learners (with a focus on 
entry-level health professionals). It was therefore appro- 
priate that the testing of the instrument compared the 
two extremes that could be present in entry-level train- 
ing of health professionals, that is, those with no expo- 
sure to formal EBP training versus those with exposure 
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to formal EBP training. While comparisons between 
novice, intermediate and expert learners would be useful 
to undertake with the K-REC instrument, it is unknown 
whether a ceiling effect might exist for the expert group, 
rendering this instrument inappropriate for this group. 

The Sicily Consensus Statement on EBP [7] high- 
lighted a need for effective training in each of the five 
steps of EBP, and future research into valid and reliable 
instruments to evaluate this training. While it seems 
reasonable in theory to advocate the need for EBP eva- 
luation instruments that cover all of the five steps of 
EBP, this may be a difficult undertaking within any sin- 
gle instrument. For example, instruments which evaluate 
cognitive EBP skills such as the Fresno test, and the 
instrument developed in the current study are not 
placed to evaluate the application of the evidence into 
practice (step 4: APPLY), or to promote reflection of 
individual performance of the entire EBP process (step 
5: ASSESS). Rather than aiming to develop instruments 
which assess all of the five steps of EBP, it may be that 
instruments are designed for specific purposes, such as 
the evaluation of EBP theoretical courses (instrument to 
assess cognitive skills), or the evaluation of EBP in clini- 
cal practice (instruments to assess performance based 
skills and application). 

Previous EBP instruments have most commonly con- 
tained items evaluating the 'ACQUIRE' and 
'APPRAISE' steps of EBP [8]. The ten instruments 
found in the updated systematic search for the current 
study (Table 1) were also consistent with this finding. 
The two instruments that were developed for occupa- 
tional and physical therapy [20,21] have closely fol- 
lowed the original Fresno test and thus evaluated 
cognitive skills pertaining to the first three steps of 
EBP. While the Sicily Statement [7] has recommended 
that instruments cover the five step process model, it 
is currently not established which core competencies 
should be included in each of the EBP steps, and 
whether these competencies are different between pro- 
fessions. For example, while it may be appropriate for 
a medical doctor involved in clinical research to 
understand how to calculate number needed to treat, 
is this a skill that an allied health professional student 
needs or should be taught during their entry-level 
training? The question of which competencies should 
be included for each of the steps in EBP evaluation 
instruments and whether this differs between profes- 
sions or levels of training (eg undergraduate or post- 
graduate) warrants further exploration. 

Conclusion 

The K-REC instrument captures cognitive EBP skills 
relating to the first three steps of the five step EBP pro- 
cess model (ASK, ACQUIRE, APPRAISE). The 



instrument has demonstrated very good reliability, and 
the validity findings show promise in the application of 
the instrument for evaluating change at an undergradu- 
ate health professional level. 

Additional material 



Additional file 1: K-REC instrument. 
Additional file 2: K-REC marking guidelines 
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